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(57) Abstract: A network usage monitoring module is provided for monitoring network usage at a network access point., i.e. net- 
woric traffic aggreganon point, typically at a gateway device or a similar network interface device. As such, the networic usage 
monitonng module can monitor the usage of a number of network users who are attempting to access various networic services pro- 
vided via the gateway device. Thus, the usage information collected by die usage monitoring module is considerably more robust 
than that offered by conventional monitoring techniques. As such, the information is considerably more valuable to network service 
providers, networic users, network beneficiaries and the like. In addition, the usage monitoring method and apparatus offers a number ' 
of particular features to improve the monitoring process as well as the value of the usage information that is coUected 
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NETWORK USAGE MONITORING DEVICE 
AND ASSOCIATED METHOD 

FIELD OF THE INVENTION 
The present invention relates generally to network usage monitoring and, 
more particularly, to a network usage monitoring device and associated usage 
monitoring mediods that operate to capture usage data at a point of netwoilc access. 

5 

BACKGROUND OF THE INVENTION 
Communication networks, such as the Internet, corporate intranets and ttie 
like, have become flie chosen form of infoimation distribution in the workplace, as 
well as, the home. With the need to access information from such networks 

10 increasing, so to is the need to provide a means for monitoring the information 
distributed by such communication networks. Monitoring of this nature 
heretofore referred to as "usage monitoring", can provide valuable infomiation, 
statistical or otherwise, to network service providers, network users or network 
beneficiaries, such as network advertisers. 

1 5 The network service providers can benefit from usage monitoring in many 

different applications; for instance, it wiU provide the capabihty to create network 
user profiles based on the mfonnation fliat the user has accessed and the 
information that the user has provided to the network. User profiles are beneficial 
to the service provider in many appUcations, such as, intelligently caching data (i.e. 

20 storing copies of network pages accessed by the user) and/or prefetching data (i.e. 
predicting future network page access and storing such). Additionally, usage 
monitoring information is instrumental in case of a network security breach. It 
provides valuable post-breach information that can be used to trace the history of 
the breaching party. 

25 Communication networks, such as the Internet, have provided another 

medium over which advertisers can distribute mformation and solicit customers. 
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Because of the vast nuinbers of people who rely upon the Ihtemet on a frequent, if 
not daily, basis, advertisers have invested enormous amounts of money to deliver 
advertisements via the Ihtemet. Unfortunately for advertisers, advertisements that 
are merely distributed on a mass basis are not typically cost effective since many, 
5 if not the majority, of the recipients have no interest in the particular product or 
service being advertised. As such, advertisers commonly work to target the 
advertisements to a subset of people who are believed to be most interested in the 
product or service being advertised. For example, new homeowners may be most 
interested in security systems and, as a result, residential security companies may 
10 target their advertising to the new homeowners. To be able to target their potential 
customer base, the advertisers must be able to access usage monitoring data in a 
real time fashion and be provided with data that is formatted, or can be readily 
formatted, to address the demographics to which their particular product is 
addressed 

1 5 Additionally, usage monitoring information can be important to Content 

Distribution Networics (CDNs) as they serve to optimize their distribution network 
on the basis of usage information and provide better services to their clients and 
the subscribers. 

Networic users, such as households or corporations, can benefit from usage 
20 monitoring information by monitoring the usage ofindividuals within the 

household or the corporation. Such monitoring insures that tihe individuals are not 
accessing inappropriate information or spending an inordinate amount of time 
browsing the Intemet or intranet. 

While communication networks, such as the Internet, provide a medium 
25 over which an enormous number of people communicate, network service 

providers, network beneficiaries and the users themselves have generally been 
unable to determine the exact nature of the users, the content of the information 
accessed by tiie users, the demographic breakdown of the users and other usage 
related information. For example, network advertisers have had difficulty 
30 determining, in real-time, both detailed demographics regarding the users and 

information relating to the actual content requested by the users. Network service 
providers have had difficulty tracking, in real time, the information accessed by 
any one user so as to provide more intelligent caching and prefetching capabilities, 
security breach investigation capabilities and the like. Additionally, the network 
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users (i.e. households or corporations) have not been provided with an adequate 
means of tracking the use of the individuals (i.e. family members or employees) in 
terms of content accessed or duration of the access session. 

In this regard, current techniques exist for monitoring network usage at 
5 different discrete locations within a communication network. For example, 
network servers and/or routers located within the stream of network 
communication can monitor usage by keepmg logs of access requests for the 
content hosted by the SCTver. These logs provide information about the nimiber of 
users that access a particular page or a set of pages and the navigational sequence 

10 throu^ the pages. Software is also provided for performing statistical processing 
of the server logs. Likewise, usage can be monitored at the user level, i.e., at an 
individual's computer, by keeping, in memory, a log of the content accessed by the 
user. This log provides information about the usage pattems of the individual user 
including the content accessed, the navigational sequence through the content and 

15 the amount of time spent on each page. Further, some Ihtemet service providers 
OBPs) can include a proxy-caching server that stores tiie most frequently requested 
information. These proxy-caching servers can be designed to monitor the web 
content usage of those users that are routed to the network via the proxy-caching 
server. Uiifortunately, proxy-caching servers require user intervention by a 

20 network administrator or the like m order to appropriately configure the proxy 
cache. Further, not nearly all client requests are directed to the proxy-caching 
server, thereby rendering the usage data incomplete and much less valuable. 

As will be noted, each of these prior techniques is associated with and 
monitors the usage of a discrete device or portion of the web. For example, 

25 monitoring that is conducted at the server level and at the user level obviously only 
identifies usage occurring at tiie server and by the user, respectively. In addition, 
monitoring conducted via a proxy-caching server only identifies some of the cUent 
requests for a particular ISP network. While these various types of usage 
monitoring are of some assistance to network service providers, network 

30 beneficiaries and network users^ these entities desire usage monitoring on a much 
broader scale covering many different users attempting to access information 
hosted by many different servers, potentially on the networks of many different 
service providers. 
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SUMMARY OF THF. TNVENTION 
A network usage monitoring module is therefore provided for monitoring 
network usage at a network access point, i.e. network traffic aggregation point, 
typically at a gateway device or a similar network interface device. As such, the 
5 network usage monitoring module ofthe present invention can monitor the usage 

of a number of network users who are attempting to access various networic 
services provided via the gateway device. Thus, the usage information collected 
by the usage monitoring module ofthe present invention is considerably more 
robust than fliat offered by convoitional monitoring techniques. As such, the 
10 information is considerably more valuable to network service providers, network 
users, network beneficiaries and the like. In addition, the usage monitoring method 
and apparatus ofthe present invention offers a number of particular features to 
improve the monitoring process as well as the value ofthe usage information that 
is collected. 

15 In one embodiment, the us^e monitoring method and apparatus captures 

all bytes of data tainsmitted between user and network services and records usage 
information associated with network addresses (i.e.. Uniform Reference Locations 
(URLs)). Typically, the usage monitoring method and apparatus will limit the 
storage of information to instances wha» a user gains access to flie network 
20 service and discards any usage information associated with addresses that are 
determined to be inaccessible, thereby reducing the size ofthe resulting database 
and improving subsequent searching ofthe database and caching ofthe database. 
The usage monitoring method and apparatus also permits the monitoring of 
streaming content in real time and facilitates the implementation of web-based user 

25 surveys. The usage monitoring method and ^paratus of the present invention 

further allows for the monitoring of user query strings as well as information input 
by a user into a web form. 

Importantly, the usage monitoring method and apparatus ofthe presait 
invention captures navigational sequaices that include one or more designated 

30 network addresses. In this regard, the designated address can either be designated 
in advance or determined on the fly by the usage monitoring module to be the most 
popular addresses. Alternatively, the usage monitoring method and apparatus can 
monitor the addresses that are in a neighborhood of a designated network address 
regardless ofthe sequence or order in which the addresses are accessed. In a 
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wo 01/86877 

similar fashion to the technique by which the usage monitoiing method and 
^paratus of the present invention monitors the addresses in the neighboihood of 
the designated address, the usage monitoring method and apparatus can monitor 
the domains that proceed and/or follow a designated domain. Furthennore, the 
5 usage monitoring method and apparatus can monitor an associated address that is 
in the neighborhood of a designated address, i.e., an associated URL that has 
proceeded or followed a designated address by no more than a predeteraiined 
number of addresses, regardless of the particular navigation sequence. The 
associated address can be defined in many ways and is typically defined by a user 

10 or network administrator to identify other associated addresses. 

By capturing a more robust set of information related to the usage of the 
communication network, the usage monitoring method and apparatus of the present 
invention can provide very useful information to usage monitoring information 
clients. For example, the usage monitoring method and apparatus can include 

1 5 detailed user demographics and information relating to the content of the accessed 
sites or of native streaming media. In addition, by monitoring the usage at the 
gateway to the Intemet, i.e., the point of trafSc aggregation, the usage monitoring 
method and apparatus can monitor the usage of a large numba: of users requesting 
access to many different service providers. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the usage monitoring apparatus according to 
one embodiment of the present invention. 

Figure 2 is a block diagram of the usage monitoring apparatus according to 
another embodiment of the invention. 
25 Figure 3 is a flow chart illustrating the general operations performed by the 

usage monitoring method and apparatus of one embodiment of the present 
invention. 

Figure 4 is a flow chart illustrating the navigational sequencing operations 
performed by usage monitoring method and apparatus, in a,ccordance with another 
30 embodiment of the invention. 

Figure 5 is a diagram illustrating a relational database that could be utilized 
by one embodiment of the usage monitoring method and apparatus of the present 
invention. 

-5- 
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nPTAH.FD DESCRI PTION OF THE INVENTION 
The present invention now will be described more fully hereinafter with 
reference to liie accompanying drawings, in which preferred embodiments of the 
invention are shown. This invention may, however, be embodied in many different 
5 forais and should not be construed as limited to the embodiments set forth herein; 
raflier, these embodiments are provided so that this disclosure will be thorough and 
complete, and will fully convey the scope of the mvention to tiiose skilled m the 
art. Like numbers refer to like elements throughout. 

A network 10 implementing usage monitoring, in accordance with the 
10 present invention is depicted in Fig. 1 . The usage monitoring module 12 is 

preferably implemented in or disposed in communication with a gateway device 
14. For example, in an embodiment in which the usage monitoring module is 
implemented in a gateway device, the gateway device can be a universal subscriber 
gateway provided by Nomadix, Me. of Westlake Village, California and described 
15 in U.S. Patent AppUcation Serial Nos. 08/816,174, 09/458,602, 09/458,569, 

09/541,877 the contents of which are incorporated by reference as if setforth fully 
herein. Altematively, the gateway device can be any of anumber of other gateway 
devices capable of implementing programming modules as known to those of 
ordinary skill in the art. It should be noted that while implementation in 
20 conjunction with a gateway device is preferred, the usage monitoring module could 
be implemented with other devices or be an independent device so long as the 
module or device implementing the module is positioned in the network so that it 
receives all requests/queries from a plurality of users and receives all responses 
coming from all the network services that comprise the network (i.e., a point of 
25 network traffic aggregation), hi one embodiment ofthe invention, the gateway 
device implementing the usage monitormg module will be positioned within the 
network between a plurality of user devices 16 and the remainder ofthe 
communications network (i.e. router 18, service provider complex 20 and Internet 
22). Thus, the usage monitoring module is implemented at a network point of 
30 access that serves as a point of traffic aggregation for all network services that are 
included within the communications network. 

The network users can communicate with the gateway device via a number 
of different user devices 16, such as handheld communication devices, portable 

-6. 
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communication devices (i.e. laptops), personal computers or the like, which are in 
commimication with the gateway device via an access multiplexor 24. 
Additionally, depending on the means of communication between the user and the 
netwo±, the network architecture may include a network interface device (NID) 
5 (not shown in Figure 1) that serves to link the various means of communication, 
such as DSL, cable, LMDS (Local Multipoint Distribution Services), dial-up, 
leased line, wireless or the like. While Figure 1 depicts the conomunication 
networic users communicating with the network througlh a single access 
multiplexer, in a typical communication network multiple access multiplexers will 

10 be impl^ented, as well as multiple network interface devices, to allow for 
expanded user capabilities. 

The gateway device 14 typically permits the users to communicate with a 
variety of IP network services, offered either by the ISP (Internet Service Provider) 
or an enterprise network (i.e. corporate intranet or the like). As shown m Figure 1, 
- 1 5 the gateway device may be in coromunication with router 18 that serves to route 
network conmiunication to the service provider server complex 20, the Litemet 22 
or any other network service (not shown in Figure 1). While Figure 1 depicts a 
single router, in a typical coromunications network, multiple routers and/or 
switching devices will be in communication with the gateway device for properly 

20 routing network conmiunications to their addressed destination. 

According to the present invention, the usage monitoring module, captures 
all packets transmitted between the users and the network services. Filtering is 
performed to extract all relevant packets that the usage monitoring client prefers to 
monitor (such as HTTP requests and responses, DNS requests and responses, etc.) 

25 and forwards such to an associated usage monitoring database 30. The requests by 
a user for network services and the responses from the network services are 
captured by the usage monitoring database. The usage monitoring database can be 
eitiier internal to the device implementing the usage monitoring module or external 
to the device so long as the usage monitoring database is communicably connected 

30 to the usage monitoring module. In one embodiment of the invention, the usage 
monitoring database stores information associated with ttie HTTP requests and 
responses and processes the information as described hereinafter. 

Figure 2 depicts a schematic representation of a network implementing 
usage monitoring having a series of databases, in accordance with an embodiment 
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of the present invention. As a result of its position at a point of traffic aggregation 
in the network architecture, the gateway device 14 receives requests ftom a nmnber 
of subscribers for Internet or other network services, hi this regard, the usage 
monitoring module implemented in flie gateway device is capable of capturing all 
5 of the packets since the gateway device operates in a promiscuous mode. Li the 
Figure 2 embodiment the usage monitoring database is iniplemented by a usage 
monitoring database server 32 ±sA has access to auxiliary databases associated 
with the usage monitoring process. 

The usage monitoring module 12 will capture packets and route them to 
10 their respective database, in accordance with one aspect of the present invention. 
If the captured packet represents a request or query from the user, the request is 
stored in the request database 34. In addition to capturing user request packets, the 
usage monitoring module will c£5)ture packets representing responses and other 
messages transmitted by the network services. In this regard, the usage monitoring 
15 module will match the responses with their respective requests. If the user request 
or query is successftd, i.e. the network service grants access or otherwise responds 
to the query, then the response is matched with the user request and the 
information pertaining to this matched set of request/response is forward to either a 
temporary database 36 or the primary database 38. If the response from the 
20 network service indicates that a valid response is not forfhconmig, such as access is 
denied, access is unavailable, a service error has occurred, etc. or if a time-out 
occurs before a response is received, then the request database deletes the 
associated request from the database. In a similar fashion, if the response from the 
network service can not be matched with a user request or if the response occurs 
25 outside of a predetermined allowable time window, then the response from the 
network service is discarded. This aspect of the invention, which provides the 
capability to discard requests that can not be matched with valid responses and vice 
versa, allows the usage monitoring apparatus to conserve storage, thereby 
minimizing the storage space required to support the usage monitoring apparatus. 
30 By not permanently storing requests/queries that do not result in successftd 

responses, the primary database 38 can intelligently provide superior performance 
in the searching/caching operations. 

Once the usage monitoring module has matched the user request to the 
network service response, the information found in the requests and responses is 



wo 01/86877 



PCT/USOl/14493 



Stored in either the temporary database 36 or the primary database 38. The usage 
monitoring module will determine if the user request is one of a pre-determined 
number of initial requests to a particular network service, such as a web page. If 
the determination is made that it is within an initial request or the pre-determined 
5 number of initial requests, then the information found in the request and the 
response is stored in the temporary database. Once the network service has been 
accessed more than the pre-detemrined number of times, the information is 
transferred to and stored in the primary database 38 and subsequent 
requests/responses are proxied to and stored in the primary database. The 

10 implementation of the temporary database accounts for the fact that many network 
services are only requested a single time or a minimal amount of times and, 
therefore, the statistical presence of these minimal use requests/responses may not 
be required when processing of the usage data information occurs. In this regard, 
the primary database is minimized so as to be more accommodating to subsequent 

15 processing of the database by eventual usage monitor information cKentsXi.e., 
network service providers, netwoik beneficiaries or networic users). The usage 
monitoiing module can be configured so that the predetermined numbervof initial 
requests is selected to balance the competing goals of recording all activity of a 
user and streamlining the database. In some cases, however, the predetermined 

20 numberissetto L ' 

Additionally, the network usage monitoring apparatus of the present 
invention may include a database of designated network addresses 40. The 
designated network addresses are typically firequently visited network addresses 
that are predefined by the network usage monitoring administrator, typically the 

25 gateway device administrator or the like. The designated network addresses are 
most often addresses that the usage monitoring information client deems 
particularly important. The usage monitoring module allows for navigational 
sequences to be stored in the database of designated network addresses for those 
network addresses predefined as "designated". A navigational sequence will 

30 typically include a predefined number of immediately preceding and/or 

immediately succeeding network addresses accessed by the user. For example, a 
network beneficiary, such as a commercial advertiser, may desire that a particular 
Internet site be designated for usage monitoring purposes. As such, when a 
network user accesses the designated Internet site, the usage monitoring module 
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will recognize the network address as being designated and will record and store 
fte preceding and succeeding navigation sequence leading up to and following the 
access of the designated networic address. It should be noted that the navigational 
sequence could extend beyond a single user session provided the interval between 
sessions is witiiin a predetomined session window. 

In one embodiment of the invention the usage monitoring apparatus is in 
communication with aprocessing engine 42 (i.e. database ftont-end), such as a PC 
or the like. The processing engine is in communication with the usage monitoring 
database and accesses the database in conjunction with a GUI (Graphical User 
Interface) implemented on the processing engine. The processing engine accesses 
the database in accordance with the commands of the usage monitoring cUent to 
process the usage monitoring data according to specific user requirements. In one 
embodiment of the present invention the processing engine is in communication 
with an insertion server 44 (shown in Figure 2). For example, the insertion server 
may comprise a targrted advertisement insertion server or a targeted survey 
insotion server. The processing aigine hnplements the necessary conmiands to 
determine what advertising or survey would benefit the network provider or ibs 
networic beneficiary based upon collected usage monitoring data. Once a'targeted 
advertisement or survey determination is made a command is sent to the msertion 
20 servCT and an appropriate advertisement or survey is forwarded to tiie network user 
via communication with the gateway device 14. The advertisement or survey will 
typically be in the form of a pop-iq) control panel sent to the user's communication 
device. 

Figure 3 illustrates a flow diagram of a method for usage monitoring, in 
25 accordance with an embodiment of flie present invention. At step 100, the usage 
monitoring module, typically implemented on a gateway device, reads all the bytes 
of information that are bang transmitted fi:om tiie plurality of users who access the 
network via the gateway device and the bytes coming Smm fhs netwoA services. 
The gateway device will typically operate in a promiscuous mode to insure that all 
30 bytes of information, regardless of destination address, are processed by the usage 
monitoring module. At step 110, the usage monitoring module detects the start of 
an HTTP (HyperText Transfer Protocol) header in the TCP (Transmission Control 
Protocol) flow. 
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The usage monitoring module may be optionally designed so as only to 
monitor the usage of network users who have accessed more than a predetermined 
number of different network addresses, such as Internet URL's (Uniform Resource 
Locator) during a session. This provides for a reduction in the database without 

5 significantly impairing the integrity of the data because users who only access a 
minimal number of addresses may not provide vital statistical data to the usage 
monitoring client. Therefore, at optional step 120, a determination is made as to 
whether the current session flow for the user has exceeded the predetermined 
threshold Tm'TiiiniiTn . If the user has not accessed more than the predetermined 

10 nimiber of URL's during a session, the usage monitoring module continues to 
monitor, at step 110, the requests transmitted by the user, but does not further 
analyze and store the requests until the number of requests exceed the 
predetermined threshold. 

Once the user has issued more than the predetermined number of requests 

15 during a session or if the usage monitoring.module does not implement this 

optional filter, then, at step 130, the usage monitoring module determines if the^^ 
packets represent a request or a response, typically by analyzing the HTTP header 
of llie packet. If the captured packet represent a request, then at step 140, the 
request is stored in the request database. If the c^tured packet represents a 

20 response, then at step 150, the response is checked to determine if the status code 
in the response is a valid or invalid status code. 

Usually, while monitoring network service accesses or other requests, Hhe 
fact that the user might not have been able to access the service for some reason (it 
may have been a dead link or the server may have been down temporarily) is not 

25 taken into account. In one aspect of the invention, however, the response status 
codes from the requested network service are also monitored in order to record 
only the addresses of the network services that the user actually gains access to 
with the other requests being deleted from the database. This can result in a 
significant saving in storage and processing computation. 

30 The status code is a three-digit integer number returned by the requested 

server. The first digit ofthe status code represents the class of the response. Some 
of the more common status codes returned are: 

1) 200 OK 

2) 301 Moved Permanently 
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3) 304 Not Modified 

4) 401 Unauthorized 

5) 403 Forbidden 

6) 404 Not Found 

5 7) 500 Mtemai Server Error 

8) 501 Not Implemented 

While a status code of 200 indicates Aat the web page can be visited, the 
other status codes indicate that the web page can not be visited If the response 
status code is determined to be invalid, i.e. a status code other than 200, then, at 
10 step 160, the response is discarded and the matching request is deleted firom the 
request database. Additionally, a user request may be deleted firom the request 
database if a predetennined timeout pmod is exceeded prior to receiving an 
associated network response. If the response status code is detemiined to be valid 
then, at step 170, the network response is matched with the associated user request 
15 , in the request database. If no match is found for the returned network response 
such as in instances in which an excessive amount of the time passed between the 
request and the response, then, at step 180, the network response is discarded. 

At step 190, the usage monitoring method and apparatus optionally 
determines if the matched request/response is in the primary database or the 
20 temporary database. While all requests/responses and information associated with 
such can be stored in the primary database, the usage monitoring method and 
apparatus of one aspect of the present invention initially, and optionally, stores 
data in a temporary database if the requested network service has been requested 
for the first time or, if configured as such, requested for less than a predetermined 
25 number of times. Thereafter, the request/response irifonnation is traiisferred to 
and/or recorded in a primary database once the network service is requested for 
more than the predetermined number of times. As previously discussed, since 
many of the network addresses, e.g. Internet URL's, are only requested a single 
time the size of the primary database can be minimized while continuing to store 
30 all of the network service requests/responses that are visited more than once by any 
user. This can be accompUshed by establishing a separate, albeit temporary, 
database for network services that are infrequently requested less than a predefined 
minimum number of tunes. 
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Thus, if no entry is found in both the permanent database and the temporary 
database (or if the minimum threshold limit has yet to be exceeded for entries in 
the temporary database) then, at step 200, the request/response info is stored in the 
temporary database. If an entry is found in the pemianent database then, at step 

S 210, the permanent database is updated to indicate a new request/response entry. 
Additionally, if an entry is found in the temporary database and the current access 
qualifies it for inclusion in the permanent database (i.e., the predetermined 
miTnimiTTi threshold limit is being exceeded) then, at step 210, the entry or entries 
from the temporary database are transferred to the permanent database. 

10 In one embodiment of the invention, the usage monitoring database stores 

the URL of the request and associated response. In general, a URL will include the 
protocol prefix, the port number, the domain name, subdirectory names and a file 
name. In alternate embodiments of the present invention, the usage monitoring 
database will preferably store, in the primary database, a variety of other data 



15 related to &e request and the response. In this regard, the following data is 
typically coUected firoin the request/response and stored in the ^propriatei 
database: 



25 d) Referring address (obtained firom the HTTP header) -typically, the 



20 



a) Subscriber Identification (This identification may be anonymized 
before being provided to the usage monitor client by mapping it to a 
random number and Avill be indexed to user profiles that provide 
information about the user's age, gmder, income, etc.) 



b) URL (URL's will be logically arranged in clusters) 



c) Timestamp - indicating the time at which the packet was processed by 
the usage monitoring module. 



address that provided &e hyperlink to the accessed address. 

e) ContCTt-type(typicaUy obtained from &e HTTP header) 

f) Content length 

g) Response Status Code 



30 



h) User Query String (usually sent as an argument to a CGI (Common 
Gateway Interface) application or as a result of the network user posting 
a form). 
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The preceding listing is not to be construed as exhaustive, other 
information may also be extracted for the requests/requests as is dictated by the 
£^lication for which the information will ultimately be used. 

While the information included in the user requests and network responses 
5 can be stored in different manners, the primary database of one embodiment of the 
invention is a relational database as depicted in Fig. 5, which relates the users and 
respective addresses/URLs. A relational database is a well-established way of 
storing data sets that may have relationships with each other. In this instance, each 
network user and each accessed address are stored one time only instead of both 
10 items being stored every time a network user accesses the address. Each access of 
a page can then be represented as a node (referred to as a tuple in the context of 
relational databases) that joins the entry of the user accessing the address and the 
entry of the address being accessed. This node will ihsa include additional 
information related to that access/accesses such as time of access, frequency of 
15 access, content length and type, etc. For example, in the instance in which user 1 1 
accesses addressAJRL 25 the relational database stores user 11 in Table 1 and 
address/URL in T2d)le 2 with a node linking the entry from Table 1 andtheentry 
from Table 2. This form of data storage provides for ease in data retrieval, such 
that, the usage monitoring client can access all the nodes that a user is linked to in 
20 order to compile information. 

In accordance with the flow diagram of Figure 4, an altemate embodiment 
of the present invention is diagramed that entails usage monitoring in the form of 
navigational sequence capturing. Network service providers and network 
beneficiaries, such as networic advertisers, are interested in knowing not only what 
25 network services are most frequently accessed and how much time is spent 

accessing them but also the navigational sequences that include these frequently 
accessed addresses. The network administrator and/or the gateway administrator, 
at the bequest of the service provider or network beneficiary, may predefine or 
"designate" specific network addresses for navigational sequencing monitoring. 
30 The administrator will store the designated network addresses in the designated 
network address database. The navigational sequence related to a designated 
network address will comprise a predefined number of network addresses that were 
accessed by the user immediately preceding and/or succeeding the access of the 
designated network address. It should be noted that while a typical navigational 
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sequence will occur during a single user session, a navigational sequence may 
extend beyond the single user session provided the interval between two sessions is 
under a specified threshold. Additionally, a designated network address may be 
part of more than one navigational sequence; therefore, the usage monitoring 
5 module of the present invention may capture and store the following information: 

■ All the navigational sequences the designated network address is a part 
of. 

■ Only the most popular navigational sequences (which exceed a certain 
threshold of visits) that a network address is part of 

10 ■ The frequency of a particular navigational sequence that the designated 

netwoik address is a part of 
Configuration of the navigational sequence database, typically included in the 
primary database will be at the option of the network administrator or gateway 
administrator. 

15 As known by those of ordinary skill in the art, Zipf s Law states that the 

number of occurrences of a word in a suf&ciently long stretch of text is flie 
reciprocal of the order of the frequency occurrence. For example, the 10^ most 
frequent word occurs exactly 10 times less frequently than the most frequent word. 
While a pure Zipf s law relationship does not readily occur, power laws (where the 

20 relationship between the order of &e frequency of occurrence and the frequency of 
occurrence is not linear) exist in a wide variety of situations, such as network 
service access frequency. One consequence of this is that a small number of 
network addresses accounts for most of the user accesses (e.g. 5% of the available 
network addresses account for 95% of network accesses). These assumptions 

25 imply that statistical data, i.e. navigational sequencing and the like, for only a small 
percentage of network addresses (e.g. 5%) needs to be readily captured and stored 
within the usage monitoring database. In this regard, the netwoik service 
providers and/or the network beneficiaries are provided the option of either 
predefining the designated network addresses that will be navigational sequence 

30 monitored or allowing the usage monitoring module to navigational sequence 
monitor the most frequently accessed network addresses as defined by the 
database. 

Referring again to the flow diagram of Figure 4, the usage monitoring 
module of the present invention monitors the navigational sequences of the 
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designated network addresses in, generally, real-time. It should be noted that steps 
100-170 of the Figure 4 flow are identical to the steps in the Figure 3 flow diagram 
and, therefore, further discussion of these steps wfll not be necessary. Navigational 
sequence monitoring is accomplished according to the present invention by 
5 keeping track of the previous k network addresses accessed by each user in their 
current session. The number k specifies the number of network addresses 
preceding and succeeding the designated network address. The k value can be 
specified by the network or gateway administrator and can diflFer over different 
designated network addresses. Although this description indicates that the same 
1 0 nimiber of network addresses are monitored both before and after the designated 
network address, a different value can be assigned for the preceding and 
succeeding network addresses, if desired. 

At step 220, the usage monitoring module deteraiines whether the network 
user has accessed a designated netwodc address in the previous k network service 
IS accesses. If a detemiination is made that the network user has accessed a 

designated network address in the previous k accesses then, at step 230, the current 
accessed network address is then added to the navigational sequence being 
constructed. If a determination is made that the network user has not accessed a 
designated networic address in the previous k accesses then, at step 240, the usage 
20 monitoring module determines if the current accessed network address is a 
designated network address by attempting to locate an entry in the designated 
network address database. If the usage monitoring module determines that the 
current accessed network address is a designated network address then, at step 250, 
a command is sent to store the previous k network addresses accessed by the user 
25 as a navigational sequence. If the usage monitoring module determines that the 
cuirent accessed network address is not a designated network address then, at step 
260, the cuirent accessed network address is added to the list of ^mox network 
addresses for that particular user and the oldest network address in the kmax hst is 
discarded if the kmox list aheady has k network addresses. The kmax list is stored 
30 because it is not known beforehand v/hich network addresses will be accessed by 
the network user and, thus, the k^ax list provides for a hsting of preceding network 
addresses if the network user accesses a designated network address and a 
navigational sequence is required. 
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In addition to monitoring user access requests and network service 
responses, the usage monitoring method and apparatus of the present invention can 
monitor usage of streaming content, in accordance with yet another embodiment of 
the present invention. The streaming content can either be live or recorded and the 
5 capture will occur, as the streaming content is delivered from the native streaming 
media server to the network user via the gateway device. Streaming content is 
typically high volume content so it is not possible or practical to store the 
infoimation related to all streaming content data packets. Therefore, while all 
streaming content packets are captured at the usage monitoring module a 

10 significant portion of the packets will be filtered out to account for this high traffic 
volume. Constant monitoring of the streaming content packets will allow for 
significant filtering to occur and will provide infoimation pertaining to the amount 
of data transferred during the streaming connection, the time duration of the 
streaming connection, etc. For this constant monitoring to occur the device ^ 

1 5 implementing the usage monitoring module, typically a gateway device, will need 
to allocate a certain amount of state in memory to keep track of the current or last- 
known status of the streaming content. Additionally, the usage monitoring ' 
module or, alternatively a post processing application, will track users currently 
accessing streaming content, the site &om which the streaming content is coming 

20 fix)m, the amount of data transferred during a connection and the last time a r 
streaming content packet was observed for a particular user/streaming content.pair. 
Since there is no explicit connection termination in the case of streaming content, 
the usage monitoring module will need to implement a timeout to determine when 
the content steaming tenoinated. These tracking fimctions can occur at the usage 

25 monitoring module or the unformatted data can be sent on a per streaming packet 
basis to a post processing application to determine the tracking parameters. 

In yet another embodiment of the present invention, the usage monitoring 
apparatus can be used to provide targeted surveys to network users. As previously 
discussed an msertion saver may be implemented to send targeted surveys to 

30 network users based upon usage monitoring data. The surveys may be sent to the 
network user unsolicited or the network user may choose to participate in the 
surveys. In this regard, network users may be selected randomly and a pop-up 
screen will appear at randomly selected times on their communication device. See, 
for example, U.S. Patent AppHcation Serial No. 09/541,877, entitled "Information 
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And Control Console For Use With A Networic Gateway Interface" ffled April 3, 
2000, in the name of inventor Short et al., and assigned to the same assignee as the 
present invention. The contents of the '877 application are hereby incorporated by 
refisrence as if setforth fully herein. 

In an alternate embodiment of the present invention, the usage monitoring 
method and apparatus may also monitor queries that a network user submits to a 
network service application, such as a search engine or the like. By monitoring liie 
query a netwo± user submits to a network service iqyplicatioii such as a search 
engine, additional knowledge about the user's preferences can be obtained. Current 
methods of usage monitoring do not provide for the cJ5)ability to monitor, in real- 
time, network user queries. Real-time monitoring of networic user query strings 
provides network beneficiaries, such as network advertisers and the like the ability 
to profile the network user and provide the user with more targeted information. 
For instance, if a user searches for the model Taurus™ on the Ford™ website, a 
network advertiser can use this information to direct an advertisement for a 
competing product to the user. 

Anotiier form of usa: input -w^ich can be monitored by the usage 
monitoring method and apparatus are the web forms posted by a user, such as 
network service registration forms, application forms and the like. Current 
20 methods do not capture information in forms that are submitted by the user on the 
web. 

As previously stated, the network usage monitoring apparatus and mefliod 
of the current invention provides for the capabiUty to monitor and store a wide 
spectrum of network user related information. This information includes, but is 

25 not limited to, the content of the network service accessed, the demographics of the 
user population (i.e. user population age, gender, geographical location, service 
plan, etc.), and the duration that network services are accessed. In this regard, the 
network service provider or beneficiary will be able to determine real-time 
statistical properties of users accessing any particular network service at any 

30 particular time. For instance, "How many women between the ages of 34 and 55 
are currently accessing a specific web-site?" or "What proportion of users listening 
to a particular classical music station are under 35 years of age?" This information 
can then be provided to network beneficiaries, who can ejqjloit this information to 
benefit their core audience in an effective manner. 
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Main Memory Requirements 

The following discussion serves to hypothetically estimate the amount of 
main memory required to operate the usage monitoring module in accordance with 
5 the present invention. 

Assuming that the average length of an address is / bytes and the number of 
active customers is n, the amount of storage can be calculated as 

Sn (Storage in bytes) -1*2 kmax * n 
The worst case storage requirement then depends on the total number of 
10 users that access the network at this particular access point 

Sn (Storage in bytes) -l"^ 2 kmax * N 
ISP (Intemet Service Provider) studies indicate that at any time only 10- 
25% of the users are active. As a result, the typical storage requirement Sn is only 
about 25% of the worst case storage requirement Sn. 
1 5 The length of a typical packet is usually under 1 00 bytes though some 

URL*s do extend to more than twice this length; The longer URL's typically result 
from infrequent queries and as a result do not occur very frequently. Thus, for the 
purpose at hand, it can be assumed that the average value of / is 100 bytes. 

The typical length of a user session varies from 55 minutes for dial-up 
20 customers to 95 minutes for DSL customers. And since the average time spent on 
a webpage by a usct is approximately 2 minutes, it is fair to estimate that the 
number of network addresses visited by any user in one network session is usually 
of the order of 100. This gives us an upper bound on the value of k. 

Thus, if it is assumed that 128MB of RAM in the primary database is 
25 dedicated to storing navigational sequences identified according to this method and 
the value oik is 100 and / is 100 bytes, the usage monitoring module will be 
capable of monitoring nearest neighbors for 12,800 xisers. One can thus conclude 
that this method scales well in terms of the amount of state required, i.e., &e 
number of preceding and succeeding network addresses that are maintained, and 
30 the main memory requirements. 

Secondary Storage Requirements 

The number of navigational sequences stored in the primary database will 
typically increase with time as the users visit more of the designated network 
addresses through different navigational sequences. The mmiber also increases 
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with the value of ^ as the navigational sequences themselves become longer. 
There are two ways to reduce the storage required without losing valuable 
infoimation about the navigational sequences. 

■ j^ng the sequences: A tunestamp is stored with each navigational 
5 sequence. This timestamp indicates the last time the sequence was 

visited. If the sequence has not been visited for a specified time period 
it is timed out and is deleted. 

■ Eliminate infrequent sequences: Zipf s law again indicates that only a 
small proportion of sequences are responsible for most of the accesses. 

10 Thus it is possible to weed out the navigational sequences that are 

visited less than a specified number of times. 

In accordance with an altemate embodiment of the present invention, the 
usage monitoring apparatus and method may be used to capture and store the 

15 neighborhood of designated network addresses. In this embodiment, there is no 
significance given to the order in which network addresses are accessed by the 
user. The only mformation stored in the database is the network addresses that 
occur within a specified number of pages (or time) of the designated network 
address. The number of network addresses in the neighborhood defined as k, 

20 usually increases as the user accesses more network addresses. The number also 
increases with the value of k. Similar to the secondary storage requirements 
defined above there are two ways to reduce the storage required without losing 
valuable information about the neighborhood. 

■ Aging the neighborhood: A timestamp is stored with each network 

25 address entry that is part of the neighborhood. This timestamp indicates 

the last time the network address was visited. If the network service 
has hot been visited for a specified time period it is timed out and is 
deleted &om the neighborhood. 

■ Eliminate infrequent neighbors: Zipf s law again indicates that only a 
30 small proportion of network addresses in the neigjiborhood are 

responsible for a majority of accesses. Thus it is possible to delete the 
network addresses that are visited below a specified number of times 
from the neighborhood. 
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Just as the previous method describes a way to capture the network 
addresses neighboihood of a designated network address; it is also possible to 
capture the domain neighborhood of a designated domain. The advertiser can 
designate the domains that it wants the usage monitor module to monitor. The 
5 usage monitor module then keeps track of the previous k domains that a user has 
accessed at any time. When the user accesses a designated domain the usage 
monitor module stores the previous k domains accessed by the usct as well as the 
following k domains that the user will access. Again it is possible to reduce the 
storage required by aging out domains as well as eliminating infirequently visited 
10 domains. 

Capturing entire navigational sequences of which the designated network 
address and a specified list of associated network addresses are part of, may be 
considered too storage and computation intensive. In this regard, an alternative 
method is provided for that tracks an associated set of network addresses in the 

1 5 neighborhood of a designated network address, within a large timeout period T 
(this is to ensure that the storage requirements and state information are reasonable 
and implementable) and with the associated set of network addresses being . 
typically defined by the gateway or network administrator. In a network advertiser 
scenario this associated set of network addresses may, for example, belong to 

20 competitors of the designated network address. The network advertiser may wish 
to monitor if prospective customers are visiting a specified hst of associated 
addresses (such as the General Motors, Volvo, BMW or Mercedes websites), while 
browsing in the neighborhood of a designated network address (such as the Ford 
website). The usage monitoring module will provide the capability to search the 

25 navigational sequences, which include the designated network addresses and the 
associated network addresses and provide the advertiser with statistics of 
customers visiting network addresses of associated sites. 

As a result of its position at the edge of the communication network, the 
usage monitoring module according to the present invention can monitor the 

30 requests issued by a plurahty of users and the responses provided by a number of 
diffi^nt network services in order to collect a canonical set of metrics that is 
useful for network service providers, network users and network beneficiaries. 
This data can be quite robust and can include web and native streaming content as 
well as user demographics and navigation sequences. Based upon the robust set of 
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data collected from a plurality of users who are accessing a wide variety of 
network services and other entities over the Internet, the usage monitoring method 
and apparatus of the present invention can provide substantial amounts of valuable 
data to netwodc service providers, network users, network beneficiaries and the 
S like. 

Many modifications and other embodiments of the invention will come to 
minH to one skilled in the art to which fins invention pertains having tiie benefit of 
the teachings presented in tiie foregoing descriptions and the associated drawings* 
Therefore, it is to be understood fliat the invention is not to be limited to the 
10 specific embodiments disclosed and that modifications and other embodiments are 
intended to be included within the scope of the appended claims. Although 
specific temas are employed herein, they are used in a generic and descriptive 
sense only and not for purposes of limitation. 
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THAT WHICH IS CLAIMED : 



1 . An apparatus for monitoring conununication network usage 
comprising: 

a usage monitoiing module operating at a network access point that 
captures all request packets transmitted from a plurality of networic users and all 
5 response packets transmitted &om a plurality of network services; wherein the 
usage monitoring module filters the captured packets to extract usage monitoring 
data therefirom; and 

a usage monitoring database in communication with said usage 
monitoring module that receives the filtered usage monitoring data fi-om the usage 
1 0 monitoring module and stores the data associated with the user request packets and 
network response packets. 

2. The apparatus of Claim 1 , further comprising a gateway device that 
implements the usage monitoring module, wherein the gate^yay device is disposed 

IS within the network at point of network traf&c aggregation so that it receives all 
requests firom a plurality of network users and receives all responses fiom network 
services. 



3. The apparatus of Claim 1, wherein the usage monitoring module 
20 filters the captured packets to extract network addresses and the usage monitoring 

database stores the network addresses. 

4. The apparatus of Claim 3, wherein the network addresses fiirther 
comprise URLs (Uniform Resource Locators). 

25 

5. The apparatus of Claim 1 , wh^in the usage monitoring module 
filters the captured packets to extract usage monitoring data comprising at least one 
type of data chosen firom the group consisting of user identification, network 
addresses, packet timestamp, referring network address, content-type, content 

30 length, response status code and user query string. 
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The apparatus of Claim 1, wherein the usage monitoring module 



10 



15 



20 



performs navigational sequencing on captured packets so as to monitor a sequence 
of network addresses accessed by a user. 

7. The apparatus of Claim 1, wherein the usage monitoring module 
performs a status code check on netwoik service response packets to verify that a 
user has access to a requested network service address so ^t only data associated 
witii accessible network services is stored in the usage monitoring database. 

8. The apparatus of Claim 1, wherein the usage monitoiing database 
further comprises a user request database that temporarily stores user request data 
prior to receiving a response from a requested network service. 

9. The apparatus of Claim 1, wherein the usage monitoring database 
further comprises a temporary database that temporarily stores iiser monitoring 
information for network addresses that have only been accessed by a user no more 
than a predeteimined Tninimum number of tunes. 

1 0. The apparatus of Claim 9, wherein the predetemiined minimum 
numba: of times is one. 

1 1 . The apparatus of Claim 1 , wherein the xisage monitoring database 
further comprises a designated network address database that stores network 
addresses that have been designated for navigational sequencing. 

12. The ^paratus of Claim 11, wherein the usage monitoring module is 
adapted to receive the network addresses &at have been designated for 
navigational sequencing from a usage monitoring client 

1 3 . The apparatus of Claim 1 1 , wherein the usage monitoring module 
defines the network addresses that have been designated for navigational 
sequencing based on the frequency that a user accesses the network address. 
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14. The apparatus of Claim 1, wherein the usage monitoring database 
further comprises a primary database that stores the data associated with the user 
request packets and network response packets. 

5 15. The apparatus of Claim 1 , further comprising an insertion server in 

communication with the usage monitoring database that provides netwoik users 
with targeted infoimation based on netwoik user usage monitoring data. 

16. The apparatus of Claim 15, wherein the insertion server further 
10 comprises an advertisement insertion server that provides network users with 
targeted advertisements based on netwoik user usage monitoring data. 



17. The apparatus of Claim 15, wherein the insertion server further 
comprises a survey insertion server that provides network users with targeted, 

15 surveys based on network user usage monitoring data. 

18. A method for providing usage monitoring in a communications 
network, the method comprising: 

capturing all data packets being transmitted from a plurality of users 
20 and a plurality of network services at a network point of access; 

filtering the captured packets to provide for usage monitoring data; 

and 

storing the usage monitoring data in a usage monitoring database. 

25 19. The method of Claim 18, wherein capturing all data packets being 

transmitted from a plurality of users and a plurality of network services at a 
network pomt of access further comprises capturing aU data packets being 
transmitted from a plurality of users and a plurality of network services at a 
gateway device. 



30 



20. The method of Claim 1 8, wherein filtaing the captured packets to 
provide usage monitoring data further comprises filtering the captured packets to 
extract usage monitoring data comprising at least one type of data chosen from the 
group consisting of user identification, network addresses, packet timestamp, 
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refening network address, content-type, content length, response status code and 
user query string. 



21. The method of Claim 18, wherein storing the usage monitoring data 
5 in a usage monitoring database, further comprises storing the usage monitoring 
data in a temporary database prior to storing the usage monitoring data in a 
primary database. 



22. A method for providing usage monitoring in a cotmnunications 
10 network, the method comprising: 

capturing a transmitted data packet at a network point of access; 
determining if the transmitted data packet is a user generated 
request data packet; and 

storing the transmitted data packet in a first database if the 
IS transmitted data packet is determined to be a user gen^ated request data packet 

23. The method of Claim 22, wherein capturing a transmitted data 
packet at a network point of access furflier comprises capturing a transmitted data 
packet at a network gateway device. 

20 

24. The method of Claim 22, wherein storing the transmitted data 
packet in a first database if the transmitted data packet is determined to be a user 
generated request data packet further comprises storing the transmitted data packet 
in a first temporary database if the trajismitted data packet is determined to be a 

25 user generated request data packet. 

25. The method of Claim 22, further comprising: 

transferring the transmitted data packet firom the first database to a 
second database if the transmitted data packet is matched with an accessible 
30 network service response data packet. 

26. The method of Claim 25, wherein the second database comprises a 
temporary database that stores the transmitted data packet and the matched 
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network service response data packet if a determination is made that the user has 
not exceeded a predetennined number of network service address accesses. 

27. The method of Claim 25, wherein the second database comprises a 
5 primary database that stores the transmitted data packet and the matched network 
service response data packet if a detemunation is made that the user has exceeded 
a predetennined number of network service address accesses. 



28. The method of Claim 22, jCurther comprising: 

10 determining if the transmitted data packet exceeds a session flow 

limit so as to warrant storage of the first transmitted data packet in the first 
database. 

29. A method for providing usage monitoring in a communications 
15 network, the method comprising: 

capturing a transmitted data packet at the network point of access, 
the transmitted data packet being determined to be a network service generated 
response packet; 

verifymg user network accessibility of the second transmitted data 

20 packet; 

matching the transmitted data packet with a user generated request 
data packet; and 

storing usage monitoring information firom the matched transmitted 
data packet and user generated request data packet in a database. 

25 

30. The method of Claim 29, wherein capturing a transmitted data 
packet at a network point of access further comprises c^turing a transmitted data 
packet at a network gateway device. 



30 



31. The method of Claim 29, wherein verifyirig network accessibility of 
the transmitted data packet further comprises checking the status code of the 
transmitted data packet to verify user network accessibility. 
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32. The method of Claim 29, wherein storing usage monitoring 
infonnation from the matched transmitted data packet and user generated request 
data packet in a database further comprises storing usage monitoring infonnation 
from the matched transmitted date packet and user generated data packet in a 

5 temporary database if a determination is made that the user has not exceeded a 
predetermined number of network service address accesses. 

33 . The method of Claim 29, wherein storing usage monitoring 
information from the matched transmitted data packet and user generated request 

10 data packet in a database further comprises storing usage monitoring information 
from the matched transmitted date packet and user generated data packet in a 
primary database if a determination is made that the user has exceeded a 
predetermined number of network service address accesses. 

15 34. A method for navigational sequence usage monitoring in a 

communications network; the method comprising: 

designating network addresses requiring navigational sequencing; 
Capturing, at a network point of access, a user transmitted data 
packet associated with a network address; 
20 determining if the network address associated with the captured user 

transmitted data packet is a designated network address; and 

storing a predetermined number of network addresses that are 
accessed proximate the designated network address as a navigational sequence if a 
determination is made that network address associated with the captured user 
25 transmitted data packet is a designated network address. 

35. The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
as a navigational sequence further comprises storing a predetermined number of 

30 network addresses that are accessed subsequent to the access of the designated 
network address as a navigational sequence. 

36. The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
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as a navigational sequence further comprises storing a predetermined mmiber of 
network addresses that are accessed prior to the access of the designated network 
address as a navigational sequence. 

5 37. The method of Claim 34, wherein storing a predetennined number 

of network addresses that are accessed proximate the designated network address 
as a navigational sequence further comprises storing a predetermined number of 
network addresses that are accessed prior to the access of flie designated network 
address as a navigational sequence. 

10 

38- The method of Claim 34, wherein storing a predetermined number 
of network addresses that are accessed proximate the designated network address 
as a navigational sequence further comprises storing a predetermined number of 
network addresses that are accessed subsequent to and prior to the access of the 
IS designated network address as a navigational sequence. 

39. The method of Claim 34, further comprising: 

determining if the user that sent the user transmitted data packet has 
accessed a designated network address in a predetermined number of previous 
20 network address accesses; and 

adding the network address from the user transmitted data packet to 
a navigational sequencing list if a determination is made that the user has accessed 
a designated address in the predetermined number of previous network address 
accesses. 

25 

40. The method of Claim 34, further comprising storing the designated 
network address in a database. 

41 . The method of Claim 34, wherein designating network addresses 
30 requiring navigational sequencing is based upon determining the most frequently 

accessed network addresses. 
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42. The method of Claim 34, wherein designatiiig network addresses 
requiring navigational sequencing is based upon network addresses accessed a 
predetermined number of times. 

5 43. The method of Claim 34, wherein designating network addresses 

requiring navigational sequencing is based upon usage monitoring client selected 
network addresses. 
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